Amendments to GOV.UK statistics publications
An exploratory data analysis to inform Reproducible Analytical Pipelines, by Duncan Garmonsway on 15 January 2019.
Introduction
The Reproducible Analytical Pipeline (RAP) is an alternative production methodology for automating the bulk of steps involved in creating a statistical report.
One motivation to use RAP methods is to avoid making trivial mistakes, by removing as much opportunity for human error as possible. For example, the following note describes a trivial amendment that could have been avoided by programming a computer to update the date of the quarterly publication.
The date for the publication … was incorrectly stated as 14 February 2019, this has now been corrected to 21 February 2019 which is in line with the release calendar and established practice to publish on the third Thursday of the month.
This article investigates.
- How common amendments are
- Whether they are becoming more or less common
- Why amendments are made The code is on GitHub
Detection of amendments
Updates are regarded as ‘amendments’ when they contain a word from a list. The list of words was compiled in discussion with colleagues and by referring to a thesaurus. Some words were discarded because there were many false positives.
The number of notes that matched each word in the vocabulary
## # A tibble: 31 x 2
## pattern n
## <chr> <int>
## 1 correct 395
## 2 amend 385
## 3 error 189
## 4 erratum 16
## 5 fix 11
## 6 mistake 4
## 7 omission 4
## 8 wrong 4
## 9 inadvertent 3
## 10 inaccuracy 1
## 11 misinterpretation 1
## 12 rectify 1
## 13 aberration 0
## 14 blunder 0
## 15 deviation 0
## 16 falsehood 0
## 17 flaw 0
## 18 glitch 0
## 19 lapse 0
## 20 misapplication 0
## 21 misapprehension 0
## 22 miscalculation 0
## 23 misconception 0
## 24 misjudgment 0
## 25 misprint 0
## 26 misstatement 0
## 27 misstep 0
## 28 misunderstanding 0
## 29 overestimation 0
## 30 oversight 0
## 31 remedy 0Searchable table of all notes
How common are changes (not just amendments)?
Many first publications were backdated. The date of the first change is a reasonable estimate for the date that GOV.UK first published statistics
Changes per month
Changes per year
How common are amendments?
The most that can be said is that “amendments happen”. The number of amendments detected is in fact quite low (about 2% of all changes), and that could be because:
- There aren’t many amendments to be detected
- Some amendments are silently fixed in a subsequent update
- The method of detecting amendments isn’t sensitive enough
On the other hand, RAP won’t necessarily reduce the number of amendments – in fact it might increase it by having greater power to detect mistakes through peer review. Errors are likely to be noticed when RAP is first applied.
Amendments per month
Amendments per year
Amendments as percentage of all changes per month
Amendments as percentage of all changes per year
Distribution of rates per organisation in the last complete year
The graph shows for each organisation:
- (Left) The total number of changes, including amendments
- (Right, ticks) The percentage of changes that were amendments
- (Right, lines) The 95% credible interval of the Empirical Bayes estimate of the rate. Longer lines show greater uncertainty about the true rate when an organisation has few publications.
Most organisations have made no amendments. Some of those haven’t published much, but a few have published a lot. Few organisations have a credible interval entirely above a 5% amendment rate.
The Empirical Bayes model was fitted with a beta-binomial prior fitted to the data by maximum likelihood estimation; see ?ebbr::ebb_fit_prior for details.
Why are amendments made?
What other changes are made?
Session info
## ─ Session info ─────────────────────────────────────────────────────────────────────────
## setting value
## version R version 3.6.1 (2017-01-27)
## os Ubuntu 16.04.6 LTS
## system x86_64, linux-gnu
## ui X11
## language en_US.UTF-8
## collate en_US.UTF-8
## ctype en_US.UTF-8
## tz UTC
## date 2019-08-13
##
## ─ Packages ─────────────────────────────────────────────────────────────────────────────
## package * version date lib source
## assertthat 0.2.1 2019-03-21 [1] CRAN (R 3.6.1)
## backports 1.1.4 2019-04-10 [1] CRAN (R 3.6.1)
## broom 0.5.2 2019-04-07 [1] CRAN (R 3.6.1)
## callr 3.3.1 2019-07-18 [1] CRAN (R 3.6.1)
## cellranger 1.1.0 2016-07-27 [1] CRAN (R 3.6.1)
## cli 1.1.0 2019-03-19 [1] CRAN (R 3.6.1)
## codetools 0.2-16 2018-12-24 [3] CRAN (R 3.6.1)
## colorspace 1.4-1 2019-03-18 [1] CRAN (R 3.6.1)
## crayon 1.3.4 2017-09-16 [1] CRAN (R 3.6.1)
## crosstalk 1.0.0 2016-12-21 [1] CRAN (R 3.6.1)
## desc 1.2.0 2018-05-01 [1] CRAN (R 3.6.1)
## devtools 2.1.0 2019-07-06 [1] CRAN (R 3.6.1)
## digest 0.6.20 2019-07-04 [1] CRAN (R 3.6.1)
## dplyr * 0.8.3 2019-07-04 [1] CRAN (R 3.6.1)
## DT * 0.8 2019-08-07 [1] CRAN (R 3.6.1)
## ebbr * 0.1 2019-08-13 [1] Github (dgrtwo/ebbr@4b9747d)
## ellipsis 0.2.0.1 2019-07-02 [1] CRAN (R 3.6.1)
## evaluate 0.14 2019-05-28 [1] CRAN (R 3.6.1)
## fansi 0.4.0 2018-10-05 [1] CRAN (R 3.6.1)
## forcats * 0.4.0 2019-02-17 [1] CRAN (R 3.6.1)
## fs 1.3.1 2019-05-06 [1] CRAN (R 3.6.1)
## generics 0.0.2 2018-11-29 [1] CRAN (R 3.6.1)
## ggplot2 * 3.2.1 2019-08-10 [1] CRAN (R 3.6.1)
## glue 1.3.1 2019-03-12 [1] CRAN (R 3.6.1)
## govdown 0.5.0.9000 2019-08-13 [1] Github (ukgovdatascience/govdown@f7f5bd7)
## gtable 0.3.0 2019-03-25 [1] CRAN (R 3.6.1)
## haven 2.1.1 2019-07-04 [1] CRAN (R 3.6.1)
## here * 0.1 2017-05-28 [1] CRAN (R 3.6.1)
## hms 0.5.0 2019-07-09 [1] CRAN (R 3.6.1)
## htmltools 0.3.6 2017-04-28 [1] CRAN (R 3.6.1)
## htmlwidgets 1.3 2018-09-30 [1] CRAN (R 3.6.1)
## httpuv 1.5.1 2019-04-05 [1] CRAN (R 3.6.1)
## httr 1.4.1 2019-08-05 [1] CRAN (R 3.6.1)
## jsonlite 1.6 2018-12-07 [1] CRAN (R 3.6.1)
## knitr 1.24 2019-08-08 [1] CRAN (R 3.6.1)
## labeling 0.3 2014-08-23 [1] CRAN (R 3.6.1)
## later 0.8.0 2019-02-11 [1] CRAN (R 3.6.1)
## lattice 0.20-38 2018-11-04 [3] CRAN (R 3.6.1)
## lazyeval 0.2.2 2019-03-15 [1] CRAN (R 3.6.1)
## lubridate * 1.7.4 2018-04-11 [1] CRAN (R 3.6.1)
## magrittr 1.5 2014-11-22 [1] CRAN (R 3.6.1)
## memoise 1.1.0 2017-04-21 [1] CRAN (R 3.6.1)
## mime 0.7 2019-06-11 [1] CRAN (R 3.6.1)
## modelr 0.1.5 2019-08-08 [1] CRAN (R 3.6.1)
## munsell 0.5.0 2018-06-12 [1] CRAN (R 3.6.1)
## nlme 3.1-140 2019-05-12 [3] CRAN (R 3.6.1)
## patchwork * 0.0.1 2019-08-13 [1] Github (thomasp85/patchwork@fd7958b)
## pillar 1.4.2 2019-06-29 [1] CRAN (R 3.6.1)
## pkgbuild 1.0.4 2019-08-05 [1] CRAN (R 3.6.1)
## pkgconfig 2.0.2 2018-08-16 [1] CRAN (R 3.6.1)
## pkgload 1.0.2 2018-10-29 [1] CRAN (R 3.6.1)
## prettyunits 1.0.2 2015-07-13 [1] CRAN (R 3.6.1)
## processx 3.4.1 2019-07-18 [1] CRAN (R 3.6.1)
## promises 1.0.1 2018-04-13 [1] CRAN (R 3.6.1)
## ps 1.3.0 2018-12-21 [1] CRAN (R 3.6.1)
## purrr * 0.3.2 2019-03-15 [1] CRAN (R 3.6.1)
## R6 2.4.0 2019-02-14 [1] CRAN (R 3.6.1)
## Rcpp 1.0.2 2019-07-25 [1] CRAN (R 3.6.1)
## readr * 1.3.1 2018-12-21 [1] CRAN (R 3.6.1)
## readxl 1.3.1 2019-03-13 [1] CRAN (R 3.6.1)
## remotes 2.1.0 2019-06-24 [1] CRAN (R 3.6.1)
## rlang 0.4.0 2019-06-25 [1] CRAN (R 3.6.1)
## rmarkdown 1.14 2019-07-12 [1] CRAN (R 3.6.1)
## rprojroot 1.3-2 2018-01-03 [1] CRAN (R 3.6.1)
## rstudioapi 0.10 2019-03-19 [1] CRAN (R 3.6.1)
## rvest 0.3.4 2019-05-15 [1] CRAN (R 3.6.1)
## scales 1.0.0 2018-08-09 [1] CRAN (R 3.6.1)
## sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 3.6.1)
## shiny 1.3.2 2019-04-22 [1] CRAN (R 3.6.1)
## stringi 1.4.3 2019-03-12 [1] CRAN (R 3.6.1)
## stringr * 1.4.0 2019-02-10 [1] CRAN (R 3.6.1)
## testthat 2.2.1 2019-07-25 [1] CRAN (R 3.6.1)
## tibble * 2.1.3 2019-06-06 [1] CRAN (R 3.6.1)
## tidyr * 0.8.3 2019-03-01 [1] CRAN (R 3.6.1)
## tidyselect 0.2.5 2018-10-11 [1] CRAN (R 3.6.1)
## tidyverse * 1.2.1 2017-11-14 [1] CRAN (R 3.6.1)
## usethis 1.5.1 2019-07-04 [1] CRAN (R 3.6.1)
## utf8 1.1.4 2018-05-24 [1] CRAN (R 3.6.1)
## vctrs 0.2.0 2019-07-05 [1] CRAN (R 3.6.1)
## VGAM 1.1-1 2019-02-18 [1] CRAN (R 3.6.1)
## withr 2.1.2 2018-03-15 [1] CRAN (R 3.6.1)
## xfun 0.8 2019-06-25 [1] CRAN (R 3.6.1)
## xml2 1.2.2 2019-08-09 [1] CRAN (R 3.6.1)
## xtable 1.8-4 2019-04-21 [1] CRAN (R 3.6.1)
## yaml 2.2.0 2018-07-25 [1] CRAN (R 3.6.1)
## zeallot 0.1.0 2018-01-28 [1] CRAN (R 3.6.1)
##
## [1] /home/travis/R/Library
## [2] /usr/local/lib/R/site-library
## [3] /home/travis/R-bin/lib/R/library